home
***
CD-ROM
|
disk
|
FTP
|
other
***
search
/
BBS Toolkit
/
BBS Toolkit.iso
/
rbbs_pc
/
mfidx12.zip
/
MAKEFIDX.DOC
< prev
next >
Wrap
Text File
|
1990-11-20
|
14KB
|
379 lines
MAKEFIDX version 1.2 - An RBBS Utility
by Ken Goosens. Nov 20, 1990
Contents
0.0 Licensing
1.0 What is MAKEFIDX?
2.0 What's New?
3.0 How to Configure RBBS-PC
4.0 How to Set Up for Files Online
5.0 How to Set Up for Files Offline
6.0 Automating Index Creation
7.0 Recompiling
8.0 An Explanation of RBBS-PC's Fast File Search
0.0 Licensing
The author does not ask for any donation for use of MAKEFIDX.
The rights to the code are exclusively retained by the author.
Everyone is free to use MAKEFIDX without any cost. Everyone is
free to distribute MAKEFIDX without cost, provided they charge nothing
for MAKEFIDX. You cannot distribute MAKEFIDX with any shareware or
commercial product without permission of the author. Modified
versions of MAKEFIDX can be distributed only with permission of the
author. Enhancements or fixes to the code are welcome. Anyone
is free to write (and charge for) an improvement over MAKEFIDX in
a language other than BASIC. For support, the author can be reached
via the RBBS conference on RBBS-Net or on his Bulletin Board at
703-978-6360.
1.0 What is MAKEFIDX?
RBBS-PC, starting with version 17.3, supports a Fast File Search that
o dramatically improves file searches on uploads and downloads
o reduces wear and tear on hard disks
o lets you have over 10,000 download areas
o increases the effective limit on the number of downloadable
files to 32,767
o introduces "do anything you want" for processing files "off line".
MAKEFIDX is a free utility for creating the index files necessary to
implement and support the Fast File Search. It should be distributed
as FIDX12.ZIP and consists of the following files:
1DISK.IMC - Macro to invoke for files listed in DISK1.DIR
ANOTHBBS.IMC - Macro informing caller on what other BBS file can be found
CHAIN.IMC - Macro objecting to files because pyramid schemes
COMMERC.IMC - Macro objecting to files because commercial
DEATHSTR.DIR - List of files on DeathStar BBS (and not here)
DEATHSTR.IMC - Macro to inform caller of files on DeathStar BBS
DISK1.DIR - List of files offline on floppy #1
MAKEFFS.BAT - Batch file to create a Fast File System
MAKEFIDX.BAS - Source code to make Fast File System index
MAKEFIDX.CFG - Configuration/driver for MAKEFIDX.EXE
MAKEFIDX.DOC - Documentation for MAKEFIDX.EXE
MAKEFIDX.EXE - Executable program to make FFS index
MAKETABS.BAS - Source code for program to make Tabs file
MAKETABS.EXE - Executable program to make Tabs file
MASTER.DIR - Sample master list of files on this BBS
OFFLINE.IMC - Macro asking if caller wants to request offline file
SUPPORT.DIR - List of files for which want special msg when downloaded
SUPPORT.IMC - Macro displaying special message for files in SUPPORT.DIR
UNWANT.DIR - List of files not wanted on this BBS
2.0 What's New
Version 1.2 is "plug compatible" with 1.1. Just replace the
MAKEFIDX.EXE. It makes the following changes:
o increased speed
Processing of filespecs is 2 to 3 times faster.
o better error trapping
Prior versions would abnormally terminate if a FileDir was
given that did not exist. Now just skips and continues processing.
o better visual feedback on progress
A count is now printed on the screen of the number of entries
processed in a FileSpec or FileDir .
o increased limits
The number of filespecs that can be handled is increased from 255
to 999.
Version 1.1 made the following changes:
(a) There is a new configuration parameter to handle non-standard
directory listings, so that MAKEFIDX can work just as well for
4DOS, or any other (see MAKEFIDX.CFG).
(b) Lists of files can now be properly processed.
Version 1.0 would not properly handle file lists that included a
drive/path in front of the file, nor lists that had only the file
name on a line, nor would 1.0 properly insert macro processing into
the location file.
These changes make it feasible to implement a system for processing
"off line" files, which is to say that download requests for files
not really present and available for downloading can still be
processed. For example, the sysop can now
o keep infrequently downloaded files on backup media, such as
tapes for diskettes, and restore these files for downloading
the next day after they are requested.
This makes it possible for sysops to make files "available", though
not immediately. Infrequently requested files can be "archived"
to backup media. No longer is it necessary to have a gigabyte of
on-line hard disks to have a gigabyte of software available.
Requests for off-line files can be logged and restorals be done
at the sysop's leisure. Room can be made on the hard disk not
by deleting files from the system but by moving them off-line
but keeping them in the directories.
o keep lists of files available on other BBS's.
Essentially, the BBS can become a central resource for informing
callers where to go to get software. Or, alternative, sysops can
design systems to automatically call other BBS's to download
requested software during the night. You can do anything you
want with requests for off-line files!
3.0 How to Configure RBBS-PC
The Fast File Search requires two parameters to be set up in
RBBS-PC's CONFIG program:
1. Parmater 267 should give the file name of list of files
avaliable for downloading. Recommended name is FIDX.DEF.
(File InDeX).
Note that the "tabs" file adds a "T" to the prefix of this
file, so the prefix should be 7 characters or fewer.
2. Parameter 268 should give the file name of the list of
locations. Recommended name is LIDX.DEF (Location InDeX).
You need not be concerned at all with the internal structure of
these files. MAKEFIDX will take care of all of the internals for
you.
However, you probably will want to change the way the download
drive/paths are configured. RBBS-PC will always search its internal
lists first, then search the file index. To realize the potential
savings in performance, you should reduce the lists of downloadable
areas to at most the upload drive/path, assuming you want uploads to
be available immediately for downloading. RBBS-PC will treat
everything in the file index to be downloadable, hence you basically
list the downloadable files by file specification in the configuration
for MAKEFIDX and not in the configuration for RBBS-PC.
4.0 How to Set up for Files Online
You should first put the files you want to be downloable by themselves
into subdirectories. You may want to sort them by subjects, but then
you will spend much time just moving uploaded files to download areas.
Most sysops just mix all the different downloadable files into different
drives and subdirectories with no need to the type of file they are.
Then you basically tell MAKEFIDX what files are available for downloading
by a standard DOS file specification, which usually consists of a drive and
path, and wildcards. For example,
C:\DWN\*.*
is all files on drive C, subsdirectory DWN. Others are:
D:\PICS\*.ZIP all files on drive D, subdir PICS, with extension ZIP
C:\RBBS\BULLET* all files on drive C, subdir RBBS, with prefix BULLET
5.0 How to Set Up for Files Off Line
The configuration option "/FileDir=" is what allows you to add files to
your lists that are off line. The feature of RBBS-PC that allows
special processing is that the location of the file (in LIDX.DEF)
supports a macro to be executed in place of a file location. The
format required is
M! <macro>
For example,
M! C:\RBBS\OFFLINE.IMC
MAKEFIDX will insert macro processing by including a line at the top
of the FileDir in the format:
Directory of M! <macro>
A good way to generate an appropriate file listing is simply to do
a DOS "DIR" with redirection to a file, e.g.
DIR A:*.* > DISK1
Then, with an editor, replace the line
Directory of A:
by
Directory of M! DISK1.IMC
In MAKEFIDX.CFG, simply have a line
/FileDir=DISK1
MAKEFIDX will put into LIDX.DEF whatever occurs after "Directory of"
and will take it to be the location of all the file names that follow
that line.
The macro file DISK1.IMC might then contain
1
{:= 2 #1
{M! OFFLINE.IMC
The second line basically sets an identifier for the appropriate back
up source to be used.
The macro in turn invoked, OFFLINE.IMC, might contain
1
{*B
The file {FI is {C1not available{C0 immediately, but is
stored {C1off line{C0 and can be {C2restored{C0 for you by
the SysOp within 24 hours.
{END
{TK
{01Request that the sysop restore the file (Y,N)
{ON 1
{==Y
{>> C:\RBBS\PLZRESTR.DAT
{FI from [2] for {FN {LN, {DT {TM
{END
{*1 Please check back tomorrow for {FI
{END ON
This simply logs to file PLZRESTR.DAT requests to store the file.
Note that the log gives with file name, disk id to restore from,
and who make the request when, using RBBS-PC's work variables and
SmartText variables.
6.0 Automating Index Creation
The process of maintaining a list of files for the RBBS-PC Fast File
Search can be automated. MAKEFFS.BAT is an example of a batch file
automating this process. The steps are:
1. Create a list of files not online but to be added to the Fast
File Search. The substeps in this process are:
a. keep a list of all of the on-line files on your system, e.g.
MASTER.DIR. Put this is the master list in CMPBBS.CFG.
b. keep lists of all additional new files to be added to the
fast file system. Set them to "AddList"s in CMPBBS.CFG.
Be sure to put in a header specifying a macro to execute.
c. Run CMPBBS to create a list of new files to add. Put this
new file list in as a "FileDir" in MAKEFIDX.CFG.
2. Set up MAKEFIDX.CFG to refer by filespec to include all
downloadable files (except those in the upload directory).
3. Run MAKEFIDX to create both a combined list of all files on
and offline, and well as a location file.
4. Sort the combined list of files (FIDX.DEF) on the file name.
Note: step 4 is set up in the batch file to use the shareware
program QSORT, which is not provided. Any good sorting program
can be used instead.
5. Run MAKETABS to produced an index on the sorted file list to
further speed up searches.
7.0 Recompiling
MAKEFIDX and MAKETABS are written in Basic for MicroSoft's
QuickBasic, and will compile in 4.5 up. Assember routines are
used from Tom Handlin's shareware Advanced Basic Library.
The commands to recompile are:
BC MAKEFIDX /O/X;
LINK MAKEFIDX,,,ADVBAS;
BC MAKETABS /O/X;
LINK MAKETABS;
8.0 An Explanation of RBBS-PC's Fast File Search
For directories DOS uses chained "buckets" in multiple locations,
searched sequentially. This results in very slow performance when
the number of files to search gets into the thousands. RBBS now
supports a fast file search than more than compensates for the
inefficent operation in DOS.
The basis for the fast file search is a file, configuration
parameter 267, which is a sorted list of file names available for
downloading. The default name is FIDX.DEF.
The format of this file is:
columns 1-12: file name
columns 13-16: location index (1, 2, 3, ...)
columns 17-18: carriage return line feed.
All data is stored as character data and the file is editable. The file
names must be stored with no internal spaces and a period separating the
prefix from the extension. The list of file names MUST BE SORTED BY
FILE NAME in order for the fast file search to work.
The location index is the record number (line number) of the locator file,
whose default name is LIDX.DEF, and has the following layout:
columns 1-63: location of file
column 64: any character. MAKEFIDX puts in a period.
columns 65-66: carriage return line feed.
This file is all character data and is editable. Essentially, the location
index points to a record in the location file. E.g. if FIDX.DEF has
RBBS-BAS.ZIP 2
HARPIE.ARC 3
and LIDX.DEF has
C:\DOWN1\
C:\DOWN2\
C:\UP\
Then RBBS-BAS.ZIP is located in directory C:\DOWN2 (2nd record) and
HARPIE.ARC is located in C:\UP (3rd record).
The location field should be a drive/path terminating with a "\" if
any path is given, and file must be filled with blanks through column
63 if the path is shorter. You must put some character in column 64.
Many editors delete trailing blanks, so you should probably put in
a non-blank. A period is a suitable choice.
RBBS will use a binary search on the first 12 characters in FIDX.DEF.
This binary search can be significantly speeded by provided "tabs" for
this file, indicating the record at which the first file is that begins
with the symbols "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ". This is like
the "tabs" you see on dictionaries, so you can directly turn to the B's,
for example. A tab file has the same prefix as the file name file,
except that it adds a "T". For "FIDX.DEF", the tab file would be
"FIDXT.DEF". RBBS will automatically detect and use a tab file if
available. The tab has 72 characters in it. Each 2 bytes represents
a binary integer whose value is the record number in FIDX.DEF where the
first file occurs that begins with the respective symbols above. Thus
bytes 3-4 show where files begin with "1" and bytes 69-70 where files
begin with "Y".